Machine Learning Methods Recap and Looking Beyond

Congratulations on completing our lesson on practical machine learning methods for 3D medical imaging analysis! The skills that you have learned through watching videos and doing exercises will help you in the final project where you will build your own AI system using a U-net implementation, and apply it to DICOM datasets.

In this lesson, we have covered the following:

• A quick refresher on how convolutional neural networks operate and a took a closer look at the different types of convolutions that underlie the operation of these networks.
• Ways to approach segmentation and classification problems for 3D medical imaging
• We did an exercise where we trained our own segmentation network on a medical imaging dataset
• Technical methods for evaluating performance of CNNs for 3D medical image analysis, and talked about the clinical aspect of evaluating performance.

Before we are ready to implement the full-scale AI solution in the final project, there is one final set of concepts that I want you to get familiar with - how to integrate such algorithms into real-world systems, and what these real-world systems look like. This would be the topic of our next lesson.

Further Resources

More problems

As mentioned in my closing remarks, machine learning problems in 3D medical imaging do not boil down to only classification and segmentation. The two problems we’ve looked at here help you understand the principles, but there is so much more you can do. Here are some pointers for some amazing things people do with deep neural networks in 3D medical imaging:

• Using deep learning to increase the resolution of low-res scans: Chaudhari AS, Fang Z, Kogan F, et al. Super-resolution musculoskeletal MRI using deep learning. Magn Reson Med. 2018;80(5):2139–2154. doi:10.1002/mrm.27178
• GANs for synthetic MRI: Frid-Adar, M., Diamant, I., Klang, E., Amitai, M., Goldberger, J., & Greenspan, H. (2018). GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing, 321, 321–331. doi: 10.1016/j.neucom.2018.09.013
• A survey of deep learning methods for medical image registration: Haskins, G., Kruger, U. & Yan, P. Deep learning in medical image registration: a survey. Machine Vision and Applications 31, 8 (2020). https://doi.org/10.1007/s00138-020-01060-x
• Overview of opportunities for deep learning on MRIs: Lundervold, A. S., & Lundervold, A. (2019). An overview of deep learning in medical imaging focusing on MRI. Zeitschrift Für Medizinische Physik, 29(2), 102–127. doi: 10.1016/j.zemedi.2018.11.002

Tools and libraries

We tried to minimize the dependency on external libraries and focus on understanding some key concepts. At the same time, there are many tools that the community has developed, which will help you get moving faster with the tasks typical for medical imaging ML workflows.

A few tools/repos worthy of attention are:

• Fast.ai - python library for medical image analysis, with focus on ML: https://dev.fast.ai/medical.imaging
• MedPy - a library for medical image processing with lots of various higher-order processing methods: https://pypi.org/project/MedPy/
• Deepmedic, a library for 3D CNNs for medical image segmentation: https://github.com/deepmedic/deepmedic
• Work by the German Cancer Research Institute:
  • https://github.com/MIC-DKFZ/trixi - a boilerplate for machine learning experiment
  • https://github.com/MIC-DKFZ/batchgenerators - tooling for data augmentation
• A publication about a project dedicated to large-scale medical imaging ML model evaluation which includes a comprehensive overview of annotation tools and related problems (including inter-observer variability): https://link.springer.com/chapter/10.1007%2F978-3-319-49644-3_4

Books

Some resources readily available online for free will help you grasp the basic concepts of computer vision and overall machine learning.

• https://d2l.ai/ - deep learning with a special section on computer vision by Alexander Smola et al. Alexander has a strong history of publications on machine learning algorithms and statistical analysis and is presently serving as a director for machine learning at Amazon Web Services in Palo Alto, CA
• http://www.mbmlbook.com/ - a book on general concepts of machine learning by Christopher Bishop et al. Christopher has a distinguished career as a machine learning scientist and presently is in charge of Microsoft Research lab in Cambridge, UK, where I had the honor to work on project InnerEye for several years.

More notable papers

• If you’re curious about segmentation space specifically, you may appreciate a foray into non-ML-based methods for segmentation. A couple of papers that can provide an introduction into that space are:
  • Boykov, Y., & Jolly, M.-P. (2000). Interactive Organ Segmentation Using Graph Cuts. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2000 Lecture Notes in Computer Science, 276–286. doi: 10.1007/978-3-540-40899-4_28
  • Probabilistic Graphical Models for Medical Image Segmentation
• This GitHub repo provides an excellent overview of CNN-based seg methods for general image domain: https://github.com/mrgloom/awesome-semantic-segmentation

Vocabulary

• Classification - the problem of determining which one of several classes an image belongs to.
• Object Detection - the problem of finding a (typically rectangular) region within an image that matches with one of several classes of interest.
• Segmentation - the problem of identifying which specific pixels within an image belong to a certain object of interest.
• 2D Convolution - an approach to feature extraction where a convolutional filter is applied to a single 2D image.
• 2.5D Convolution - an approach to feature extraction where 2D convolutions are applied independently to areas around each voxel (either in neighboring planes or in orthogonal planes) and their results are summed up to form a 2D feature map. Such an approach leverages some 3-dimensional information.
• 3D Convolution - an approach to feature extraction where the convolutional kernel is 3 dimensional and thus combines information from all 3 dimensions into the feature map.
• Longitudinal follow up - radiological analysis method that involves monitoring how things change over time. These methods are very valuable in, e.g., oncology for tracking slow-growing tumors.
• Radiation Therapy - one of the methods of treating cancer where the tumor is exposed to ionizing radiation.